Papers by R. Thomas McCoy
Evaluating distillation methods for data-efficient syntax learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | knowledge distillation (KD) targeting attention should selectively accelerate syntax acquisition, a study finds . logit-based KD dramatically improves data-efficiency, attention-based one provides minimal benefit even for syntactic tasks. |
| Approach: | a study predicts that knowledge distillation targeting attention should selectively accelerate syntax acquisition . a systolic analysis of student models compared to logit-based knowledge distillations . |
| Outcome: | a new study shows that knowledge distillation (KD) targeting attention accelerates syntax acquisition . the hypothesis is tested on syntactic benchmarks and perplexity. |
How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN (2023.tacl-1)
Copied to clipboard
| Challenge: | Current language models generate high-quality text, but are they copying it or have they learned generalizable linguistic abstractions? |
| Approach: | They propose a suite of analyses for assessing the novelty of generated text . they focus on sequential structure (n-grams) and syntactic structure (syntactical structure). |
| Outcome: | The proposed model-generated text is as novel as the baseline human-generated model- generated text, but it is copied substantially, the authors show . |
Is In-Context Learning a Type of Error-Driven Learning? Evidence from the Inverse Frequency Effect in Structural Priming (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent pre-trained large language models have shown the capacity to perform in-context learning (ICL) this capability could provide a way to bridge the divide between language models and humans. |
| Approach: | They propose a new way of diagnosing whether ICL is error-driven learning . they simulated structural priming with ICL and found the effect was stronger . |
| Outcome: | The proposed method is based on the inverse frequency effect (IFE) phenomenon is similar to error-driven learning in large language models . |
Picking BERT’s Brain: Probing for Linguistic Dependencies in Contextualized Embeddings Using Representational Similarity Analysis (2020.coling-main)
Copied to clipboard
| Challenge: | Contextualized word embeddings can incorporate contextual information, whereas other embeddables cannot. |
| Approach: | They propose an approach to address this question using Representational Similarity Analysis (RSA) they investigate whether verb embeddings encode verb’s subject, pronoun embedds antecedent and full-sentence representations encode sentence’s head word . |
| Outcome: | The proposed approach can adjudicate between hypotheses about which aspects of context are encoded in representations of language. |
Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks (2020.tacl-1)
Copied to clipboard
| Challenge: | Inductive biases can arise from any aspect of the model architecture, study finds . we investigate which architectural factors affect how models generalize . |
| Approach: | They investigate which architectural factors affect generalization behavior of neural network models . they use English question formation and English tense reinflection as test cases . |
| Outcome: | The findings suggest that human-like generalization requires architectural syntactic structure. |
Syntactic Data Augmentation Increases Robustness to Inference Heuristics (2020.acl-main)
Copied to clipboard
| Challenge: | Pretrained neural models lack sensitivity to word order on controlled challenge sets . augmentation methods that improve accuracy on standard training sets may be a problem . |
| Approach: | They propose to augment standard training sets with syntactically informative examples by applying syntastic transformations to sentences from the MNLI corpus. |
| Outcome: | The proposed method improved BERT’s accuracy on controlled examples that diagnose sensitivity to word order from 0.28 to 0.73 without affecting performance on the MNLI test set. |
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (P19-1)
Copied to clipboard
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, Samuel R. Bowman
| Challenge: | State-of-the-art models in natural language processing (NLP) often incorporate sentence encoder functions which generate a sequence of vectors intended to represent the in-context meaning of each word in an input text. |
| Approach: | They conduct the first large-scale systematic study of candidate pretraining tasks, comparing 19 different tasks as alternatives and complements to language modeling. |
| Outcome: | The proposed model can be used to train sentences on language modeling tasks. |
Representations of Syntax [MASK] Useful: Effects of Constituency and Dependency Structure in Recursive LSTMs (2020.acl-main)
Copied to clipboard
| Challenge: | a constituency-based network generalizes more robustly than a dependency-based one . a sequential LSTM can learn the appropriate rules governing these dependencies . |
| Approach: | They evaluate whether constituency-based networks introduce biases for syntactic structure . they find that constituency networks generalize more robustly than dependency networks . |
| Outcome: | The proposed model generalizes more robustly than a dependency-based model, the study shows . it shows that data augmentation can improve the robustness of the model on small data sets. |
How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speech (2023.acl-long)
Copied to clipboard
| Challenge: | LSTMs and Transformers perform well at capturing the surface statistics of child-directed speech, but both model types generalize in a way consistent with an incorrect linear rule than the correct hierarchical rule. |
| Approach: | They train LSTMs and Transformers on text from the CHILDES corpus and evaluate what they learn about English yes/no questions. |
| Outcome: | The proposed models perform well at capturing the surface statistics of child-directed speech, but generalize more consistent with an incorrect linear rule than the correct hierarchical rule. |